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Preface 


The  original  purpose  of  this  thesis  was  to  develop  a 
handbook  that  could  be  used  by  reliability  engineers  tasked 
with  including  reliability  tasks  in  an  acquisition  contract. 
However  the  Product  Assurance  Handbook  written  by  the 
Columbia  Research  Corporation  was  sent  to  my  advisor  while 
I  was  writing  this  thesis.  The  discovery  of  the  handbook 
caused  the  whole  focus  of  this  project  to  change  because  the 
handbook  completely  accomplished  the  initial  intent  for  both 
reliability  and  maintainability  tasks.  Instead  of  citing 
actual  statements  to  include  in  the  SOW,  as  originally 
intended,  the  reasons  why  these  statements/tasks  should  or 
should  not  be  included  became  paramount.  I  had  already 
changed  my  topic  once,  thus  this  refocus  was  yet  another 
stumbling  block  to  overcome. 

With  help  and  encouragement  from  my  thesis  advisor 
Captain  Clint  Campbell,  and  readers  Mr.  Roy  Wood  and  Mr. 

Mike  Schubert  I  managed  to  complete  this  project.  They  well 
deserve  and  have  my  sincere  appreciation. 

I  must  also  recognize  my  husband.  Skip,  who  kept  asking 
me  when  my  thesis  would  be  finished,  knowing  full  well  it 
would  not  be  done  until  it  was  due. 
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Abstract 

This  thesis  analyzed  the  cost  effectiveness  of  the  18 
tasks  specified  in  MIL-STD-785B ,  Reliability  Program  for 
Systems  and  Equipment  Development  and  Production.  The  pur¬ 
poses  of  the  tasks  are  described  and  each  task  was  evaluated 
according  to  six  criteria.  Cost  effectiveness  tables  were 
developed  for  Airframe/Mechanical  Equipment,  Avionics/ 
Electrical  Equipment,  and  Space  and  Missile  Systems.  The 
tables  shows  averages  taken  from  surveys  completed  by 
reliability  instructors  teaching  at  AFIT  and  reliability 
engineers  employed  by  Aeronautical  Systems  Division  and  Air 
Force  Aquisition  Logistics  Center.  This  analysis  also 
includes  a  discussion  of  the  applicability  of  the  tasks 
according  to  program  phase. 

I 

i 

I 

■ 


ANALYSIS  OF  THE  COST  EFFECTIVENESS  AND  APPLICABILITY 
OF  THE  RELIABILITY  TASKS  OF  MIL-STD-785B 


I.  Research  Problem 


Introduction 

Reliability  is  a  concept  that  has  attracted  quite  a  lot 
of  attention  recently,  not  only  in  the  military  but  through¬ 
out  industry.  Why  such  concern? 

In  the  past,  development  of  a  weapon  system  was  pri¬ 
marily  concerned  with  cost,  schedule  and  performance.  If 
these  parameters  were  met,  a  program  was  initially  con¬ 
sidered  a  success.  However  if  the  system  was  unreliable, 
(ie.  it  broke  down  consistently)  maintenance  and  spare 
parts'  costs  (life  cycle  costs)  escalated.  The  original 
acquisition  schedule  may  have  been  met  but  planned 
operational  schedules  must  be  discarded.  Performance  is 
worthless  unless  a  system  is  operating. 

Thus  reliability  is  of  considerable  interest  (or 
should  be)  to  any  organization  dealing  with  weapon  systems 
or  equipment  of  any  kind.  Reliability  of  systems  has  been 
mandated  for  some  time  by  policy  letters  originating  at  the 
Secretary  of  Defense  level  and  reinforced  at  lower  echelons 
of  command  throughout  the  Department  of  Defense  (DOD)  and 
the  Air  Force  (AF) .  The  following  quote  is  from  a  policy 
letter  dated  16  August  1974  signed  by  General  Phillips, 
Commander  of  Air  Force  Systems  Command  (AFSC) : 
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Reliability  considerations  are  as  important  to  a 
successful  development  program  as  cost,  schedule  or 
performance  and  are  equally  deserving  of  your 
attention  (7) . 

This  next  quote  is  from  the  Action  Memorandum  signed  by  both 

General  Gabriel,  Chief  of  Staff  of  the  Air  Force  and  Vern 

Orr,  Secretary  of  the  Air  Force  dated  17  September  1984: 

For  too  long,  the  reliability  and  maintainability  of 
our  weapon  systems  have  been  secondary  considerations 
in  the  acquisition  process.  ...  Reliability  and 
maintainability  must  be  coequal  with  cost,  schedule 
and  performance  as  we  bring  a  system  into  the  Air 
Force  inventory.  (10) 

An  outcome  from  the  Action  Memorandum  was  a  study 
group  and  an  HQ  USAF  action  plan  titled  R&M  2000,  dated  1 
February  1985.  The  plan's  objective  is  to  ensure  "that 
reliability  and  maintainability  (R&M)  are  considered  across 
all  weapon  systems  and  treated  equally  with  cost,  schedule, 
and  performance"  (16:1).  For  this  plan  to  succeed,  it  is 
mandatory  that  the  R&M  requirements  are  properly  stated  and 
all  relevant  specifications  are  referenced  in  the  contract. 

In  addition  to  policy  letters  over  the  years,  there 
have  been  voluminous  reports  and  regulations  covering  the 
need  for  reliability,  but  there  seems  to  be  a  problem  in 
actually  contracting  for  reliability.  In  order  to  build/ 
develop  a  useful  and  reliable  system  specific  reliability 
requirements  must  be  developed  and  communicated  to  the 
contractor . 

R&M  is  often  used  as  one  word  and  both  reliability  and 
maintainabi l i ty  affect  each  other.  However: 
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Reliability  is  a  relatively  independent  discipline 
that  should  not  be  confounded  with  maintainability. 
Reliability  is  positive.  Maintainability  is  negative, 
(except  for  preventive  maintenance) ,  a  spillover  from 
imperfect  reliability.  In  short,  if  perfect  reliabil¬ 
ity  is  achieved;  nothing  needs  to  be  repaired.  (22:25) 

Realistically,  perfect  reliability  is  almost  impossible, 
thus  maintainability  is  a  necessary  part  of  any  program. 
However  in  order  to  narrow  the  scope  of  research,  only  con¬ 
tracting  for  reliability  shall  be  the  subject  of  this  paper. 


Problem  Statement 

A  problem  occurs  when  inexperienced  people  are  as¬ 
signed  the  task  of  developing  the  reliability  criteria  for  a 
contract.  When  is  it  appropriate  to  require  the  tasks  spe¬ 
cified  in  Military  Standard  (MIL-STD)  785B,  Reliability 
Program  for  Systems  and  Equipment  Development?  How  cost 
effective  are  the  tasks?  In  order  to  answer  these  questions 
an  analysis  of  each  task  was  done  to  determine  the  task's 
applicability  in  each  acquisition  phase  and  its  cost 
effectiveness. 


Background 

The  DOD  has  defined  reliability  in  MIL-STD-721C, 
Definitions  of  Terms  for  Reliability  and  Maintainability. 
Reliability  is: 

(1)  The  duration  or  probability  of  failure-free 
performance  under  stated  conditions. 

(2)  The  probability  that  an  item  can  perform  its 
intended  function  for  a  specified  interval  under 
stated  conditions.  (11:54) 


3 


Accocding  to  this  definition  reliability  is  simply  the 
fact  that  something  works  as  intended. 

However  the  reliability  intentions  are  not  always 
stated  or  may  be  stated  incorrectly.  A  contractor  cannot  be 
obligated  to  produce  a  more  reliable  system  than  that  which 
is  specified.  Reliability  specifications  must  be  clearly 
stated  but  not  so  binding  as  to  inhibit  creativity  in 
design. 

Reliability  is  a  performance  parameter  and  needs  to  be 
designed  into  the  system.  Once  a  system  has  been  designed 
and  built,  it  is  possible  and  necessary  to  test  that  sys¬ 
tem's  reliability.  Testing  will  not  increase  reliability, 
since  reliability  will  only  change  from  a  redesign.  Testing 
is  important  however  and  needs  to  be  continuous  from  the 
beginning  of  the  system  aquisition  process.  For  testing 
produces  failures  which,  when  analyzed,  can  indicate  where 
redesign  is  necessary.  If  a  deficiency  in  reliability  is 
discovered  early  enough  it  is  possible  to  redesign  with 
minimal  cost.  Also,  screening  (the  operation  to  identify 
and  remove  bad  parts)  is  sometimes  considered  a  test  and  can 
nelp  improve  the  overall  quality  (or  reliability)  of  the 
product  by  elimination  of  failed  components. 

Reliability  is  designed  into  the  product  by  having  a 
good  Reliability  Program.  And  the  Program  is  structured  in 
the  contract.  The  original  intent  of  this  thesis  was  to 
develop  a  Contracting  for  Reliability  Primer  to  be  used  as  a 
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guide  to  help  AF  project  managers  correctly  state  in  the 
contract  the  reliability  tasks  required.  However  during  the 
course  of  this  study  the  Product  Assurance  and  Test 
Engineers  Contracting  Handbook  was  discovered.  The  Columbia 
Research  Corporation  published  this  handbook  in  1982  under 
the  auspices  of  the  Army.  This  handbook  gives  specific 
examples  of  how  to  include  the  various  tasks  of  MIL-STD-785B 
into  the  Statement  of  Work  (SOW) .  The  handbook  essentially 
covers  the  exact  area  intended  to  be  covered  by  this  thesis. 
Therefore  the  focus  of  this  project  changed  to  analyzing  the 
cost  effectiveness  of  each  of  the  tasks  and  identifying  the 
appropriate  time  in  the  acquisition  cycle  to  apply  the  task. 

Scope  of  Research 

The  scope  of  this  research  effort  is  limited  to  anal¬ 
ysis  of  the  application  of  the  18  tasks  of  MIL-STD-785B. 

This  thesis  deals  specifically  with  the  cost  effectiveness 
and  timing  of  the  application  of  the  tasks  in  an  acquisition 
contract. 

This  effort  does  not  attempt  to  cover  all  the  aspects 
of  a  reliability  program  that  are  usually  present  in  suc¬ 
cessful  management  of  reliability  of  an  acquisition  project. 
It  will  be  assumed  that  a  Statement  of  Need  and  Specifica¬ 
tion  will  have  already  been  published,  thus  reliability 
goals  will  have  been  determined  and  the  chosen  contractor 
should  be  committed  to  meeting  the  reliability  requirements. 


Methodology  and  Format 

This  section  describes  the  methodology  used  for  the 
research  project  and  the  format  of  this  thesis. 

Methodology  for  Research.  The  methodology  used  for 
this  research  encompassed  personal  interviews,  a  literature 
review,  and  an  analysis  of  the  cost  effectiveness  of  relia¬ 
bility  tasks. 

The  interviews  were  of  reliability  engineers  involved 
in  contracting  for  reliability.  These  interviews  were  un¬ 
structured  and  used  as  background  for  understanding  of  the 
many  aspects  of  contracting  for  reliability.  Relevant 
regulations  and  published  guidelines  were  used  to  determine 
necessary  aspects  of  a  contract  to  ensure  reliability.  Also 
contracts  already  in  force  were  acquired  and  used  as  addi¬ 
tional  background. 

A  cost  effectiveness  chart  was  developed  based  on  the 
"Cost  Effectiveness  influences"  table  in  Military  Handbook 
(MIL-HDBK)  338,  Electronic  Reliability  Design  Handbook  (12). 
Individuals  familiar  with  contracting  for  reliability  were 
asked  to  fill  in  the  chart  based  on  their  subjective  view  of 
the  cost  effectiveness  of  tasks  taken  from  MIL-STD-785B. 

Each  task  was  evaluated  according  to  the  six  criteria  used 
in  MI  L-HDBK-33  8  . 

Thesis  Format .  The  format  of  this  thesis  consists  of 
five  chapters  and  the  appendix.  This  first  chapter  contains 
che  introduction,  problem  statement,  background,  the  method¬ 
ology  used  in  the  research  and  the  format  of  this  thesis. 
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Chapter  II  contains  the  literature  review,  which 
concentrates  primarily  on  the  regulations,  MIL-STDs  and 
other  pertinent  reliability  handbooks  and  guides.  In 
addition  a  brief  historical  background  of  the  reliability 
efforts  by  the  military  is  also  presented. 

Chapter  III  discusses  each  task  of  MIL-STD-785B  by 
describing  the  purpose  of  the  task. 

In  Chapter  IV  the  results  from  the  cost  effectiveness 
surveys  are  presented  and  the  relative  value  of  each  task  is 
analyzed.  The  four  acquisition  phases  are  described  and  the 
applicability  of  the  reliability  tasks  according  to  program 
phase  is  discussed.  The  tasks  that  are  relevant  for  each 
phase  are  identified. 

The  concluding  Chapter  V  summarizes  this  thesis  and 
gives  recommendations  for  future  efforts  in  the  reliability/ 
contracting  area. 


II 


Literature  Review 


This  chapter  presents  the  review  of  pertinent  liter¬ 
ature  on  the  subject  of  systems  reliability  and  how  best  to 
contract  to  achieve  reliability.  Because  of  the  vastness  of 
the  literature  currently  available  on  reliability,  some  con¬ 
straints  were  necessarily  imposed  on  the  literature  review. 
Thus  this  review  deals  primarily  with  applicable  directives, 
regulations,  and  standards.  However  numerous  articles,  re¬ 
ports  and  guides  are  also  included  for  they  were  found  to  be 
quite  helpful  as  background  information. 

Historical  Aspects  of  Reliability 

A  paper  by  Thomas  Musson  titled  "A  Reliability 
Chronology"  gives  a  succinct  view  of  the  shifts  of  emphasis 
of  defense  reliability  activities  beginning  in  the  1950's. 

He  cited  the  evolution  of  the  actual  goals  or  objectives 
that  reliability  of  systems  should  achieve.  ("The  relia¬ 
bility  activities  are  only  means  to  achieve  an  identified 
end . . . reliabil ity . . . i tself  is  not  a  goal  that  is  sought.") 
(24:14) 

The  objectives  evolved  from  increasing  operational 

time,  through  enhancing  mission  performance  and  keeping  Life 

Cycle  Costs  (LCC)  down,  to  achieving  combat  readiness,  which 

Musson  feels  is  the  present  emphasis.  However  the  emphasis 

on  readiness  is  changing  to 

an  increase  in  the  attention  placed  on  the  technical 
manpower  and  skill  levels  required  within  the  defense 


establishment.  ...with  an  increased  emphasis  on 
reliability  as  a  means  of  decreasing  the  demand  for 
the  number  of  technicians  and  the  demand  for  high 
skill  levels.  (24:15) 

The  above  quote  actually  defines  supportablility,  which 
becomes  increasingly  less  costly  as  reliability  increases. 

The  shifts  of  emphasis  occurred  through  periodic 
reviews  and  examinations.  D00  Directive  5000.40,  Reliabil¬ 
ity  and  Maintainability,  published  in  1980,  was  an  attempt 
to  correct  deficiencies  and  achieve  better  reliability  by 
focusing  "more  attention  on  the  design  engineering  aspects 
of  reliability  as  opposed  to  the  numerical  aspects"  (24:16) 
During  the  fifties  the  science  of  reliability  concen¬ 
trated  on  numbers,  and  the  attempt  to  define  reliability  by 
numbers.  It  was  during  this  period  that  the  statistics  and 
the  building  of  models  for  reliability  actually  began.  In 
the  late  1960's  the  concentration  was  on  reliability 
demonstrations  through  testing.  A  problem  was  found  with 
this  type  of  testing: 

...the  reason  that  reliability  demonstrations  do  not 
guarantee  acceptable  reliability  in  the  field  is  that 
the  test  environments  do  not  simulate  the  field 
environment  (24:17). 

Thus  the  Combined  Environment  Reliability  Test  (CERT),  in 
the  early  1970's,  was  used  to  try  to  introduce  realistic 
environments  in  testing.  CERT  is  an  attempt  to  simulate 
real  environments  by  simulataneously  inducing  temperature 
cycling,  vibration,  and  changes  in  humidity. 


Another  panacea  cited  by  Musson  which  was  supposed  to 
forever  eliminate  the  problem  of  poor  reliability  was  the 
warranty,  or  more  specifically  the  Reliability  Improvement 


Warranty  (RIW)  first  advocated  in  the  1970's. 

The  RIW  plan  commits  the  contractor  to  perform 
stipulated  depot-type  repair  services  for  a  fixed 
operating  time,  calendar  time,  or  both,  at  a  fixed 
price  (12:  Sec  12,  19). 

The  objective  of  the  RIW  is  "to  secure  reliability 

improvement  and  reduce  support  costs"  (12:  Sec  12,  19).  The 

RIW  has  been  determined  to  oe  an  effective  tool  but  it  is 

"inappropriate  to  some  procurements  and  that  the  use  of  a 

RIW  would  not  guarantee  a  motivated  contractor"  (24:17). 

MIL-HDBK— 338 ,  Electronic  Reliability  Design  Handbook, 

describes  23  different  product  performance  agreements  that 

can  be  applied  to  DOD  contracts,  with  war ranties-guarantees 

agreements  being  the  most  commonly  applied  by  the  DOD 

(12:12-14  to  12-13).  As  brought  out  by  Mary  Ann  Gilleece, 

Former  Deputy  Under  Secretary  of  Defense  for  Acquisition 

Manage-ment,  all  warranties  cost  money. 

Use  of  warranties  should  be  applied  on  a  case-by¬ 
case  basis  and  should  reflect  a  balance  of  risk 
oetween  the  government  and  the  contractor  and  the 
attendant  cost  considerations  (21:28). 

Since  the  RIW  did  not  solve  all  reliability  problems, 

to  simplify  the  process  and  try  to  get  reliability  by 

design,  was  emphasized  as  another  panacea.  A  good  paper 

design  is  an  essential  element  of  a  successful  reliability 

program  but  does  not  guarantee  the  successful  performance  of 


a  product  in  the  field.  (24:17)  Today  there  is  not  one 
method  that  guarantees  reliability.  A  good  program  which 
takes  into  consideration  the  many  avenues  available  however 
does  have  a  good  chance  of  success. 

Directives,  Regulations  and  Standards 

This  section  outlines  the  directives,  regulations  and 
military  standards  (MIL-STD's)  applicable  for  any  AF 
reliability  program  and  judged  most  applicable  and/or  useful 
to  this  thesis. 

Department  of  Defense  Directive  (DODD)  5000.1,  Major 
Systems  Acquisitions  (13),  is  the  governing  directive  for 
acquiring  major  weapon  systems.  Achieving  reliability  is 
specified  to  be  included  as  a  precept  for  management  to 
apply  throughout  the  acquisition  process. 

DODD  5007.40,  Reliability  and  Maintainability  (15), 
establishes  policies  and  responsibilities  for  R&M. 

According  to  this  directive  the  major  objectives  of  defense 
R&M  activities  are  to  increase  operational  effectiveness, 
reduce  life  cycle  costs  (LCC)  and  manpower  requirements, 
provide  information  and  operate  in  an  efficient  manner.  DOD 
5000.40  directs  program  managers  and  acquiring  activities  to 
integrate  and  tailor,  allocate,  address,  and  enforce  and 
correct  deficiencies  of  R&M  engineering  tasks  and  tests. 

This  is  the  official  guideline  for  all  R&M  programs. 
Subsequent  lower  echelon  regulations  should  be  based  on 
DOD  5000.40. 


Department  of  Defense  Instruction  (DODI)  5000.2,  Major 
System  Acquisition  Procedures  (14),  implements  DODD  5000.1. 
Section  9  of  DODI  5000.2  deals  with  reliability  and 
maintainability  ( R&M) ,  stating  that  goals  and  thresholds  of 
R&M  should  directly  relate  to  "operational  readiness, 
mission  success,  nuclear  and  nonnuclear  survivability  and 
endurance,  maintenance  manpower  cost,  and  logistic  support 
cost"  (14:16).  The  instruction  also  calls  for  reliability 
growth  to  be  "predicted  and  graphically  displayed"  (14:17) 
for  reviews,  and  resources  for  design  corrections  to  be 
identified. 

Air  Force  Regulation  (AFR)  800-18,  "Air  Force 

Reliability  and  Maintainability  Program"  (5) ,  is  the  AF 

regulation  that  implements  DODD  5000.40.  It 

establishes  policy  to  ensure  every  Air  Force  system 
is  available  when  needed,  will  perform  its  assigned 
missions,  and  can  be  operated  and  supported  econo¬ 
mically  (5:1)  . 

This  regulation  primarily  assigns  r esponsioil ity  for  R&M  by 

designating  specific  requirements  and  objectives  for  each  of 

the  following  organizations: 

HQ  USAF/LE  -  Deputy  Chief  of  Staff  for  Logistics 

HQ  USAF/RD  -  Deputy  Chief  of  Staff  for  Research  and 
Development 

The  Implementing  Command 

HQ  AFSC  -  Headquarters  Air  Force  Systems  Command 
HQ  AFLC  -  Headquarters  Air  Force  Logistic  Command 
The  Operating  Command 
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AFTEC  -  Air  Force  Test  &  Evaluation  Center  (Now  known 
as  AFOTEC  -  Air  Force  Operational  Test  and 
Evaluation  Center) 

Air  Force  Systems  Command  (AFSC)  Supplement  1  to  AFR 
330-13  (6)  is  AFSC's  implementation  of  its  R&M  requirements 
designated  by  AFR  800-18.  This  supplement  details  the 
responsibilities  of  a  program  manager,  states  R&M  training 
is  required  by  engineers  assigned  R&M  responsibility  and 
goes  into  detail  on  the  many  tasks  and  tests  necessary  for  a 
viable  reliability  program.  This  supplement  also  delineates 
the  responsibilities  of  the  various  divisions  within  AFSC. 

Air  Force  Logistics  Command  (AFLC)  Supplement  1  to  AFR 
800-18  (4)  is  AFLC 1 s  implementation  of  its  R&M  requirements 
designated  oy  AFR  800-18.  AFLC's  supplement  is  a  short 
document  (only  one  page)  which  primarily  assigns  R&M 
responsibilities  to  the  AFLC  organizations  while  leaving  the 
details  up  to  those  organizations. 

Electronic  System  Division  Regulation  (ESDR)  800-5, 
Reliability  Program  for  Systems  and  Equipment  Development 
and  Production  (3)  is  an  attempt  by  a  major  product  division 
to  standardize  how  to  contract  for  reliability. 


For  most  programs  the  reliability  related  tasks  are 
stated/defined  in  MIL-STD-785B,  Reliability  Program  For 
Systems  and  Equipment  Development  and  Production  (18) .  MIL- 
STD-785B  contains  18  tasks.  Each  task  states  the  purpose, 
gives  a  description  of  what  is  to  be  accomplished  and  lists 
"DETAILS  TO  BE  SPECIFIED  8¥  THE  PA  [procuring  activity] 
(REFERENCE  1.2. 2.1)"  (18).  The  reference  refers  to  the 
paragraph  in  the  beginning  of  the  standard  which  states  that 
all  tasks  must  be  tailored  and  is  found  in  every  task.  This 
standard  is  used  as  a  framework  for  this  project. 

MIL-STD-1543A  (USAF) ,  Reliability  Program  Requirements 
for  Space  and  Missile  Systems  (17)  ,  is  the  reliability 
standard  for  space  and  missile  systems.  This  document  uses 
requirements  (rather  than  tasks)  specified  for  the  indivi¬ 
dual  phases  of  the  program  to  implement  reliability  proce¬ 
dures.  The  difference  between  this  standard  and  the  DOD 
MIL-STD-7853  is  primarily  in  the  formats.  MIL-STD-1543A 
uses  a  narrative  format  versus  the  lists  used  by  785B.  The 
same  areas  of  reliability  are  covered  in  both,  while  the 
space  requirements  are  more  specifically  designed  for  space 
systems . 

Other  Relevant  Publications 

This  section  presents  an  overview  of  various  publica¬ 
tions  found  to  be  useful  for  this  thesis  and/or  for  the 
formulating  of  a  contract. 
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(16)  includes  six  major  objectives: 

I.  Establish  clear  direction  for  R&M  improvement 
through  visible  goals  and  policy  to  increase  combat 
effectiveness  and  operational  supportability . 

II.  Establish  an  organizational  infrastructure  to 
implement  the  essential  elements  of  the  R&M  improve¬ 
ment  program,  to  form  a  base  of  technical  expertise, 
and  to  build  advocacy,  authority,  and  accountability 
into  the  R&M  program. 

III.  Establish  an  R&M  planning  system  to  consolidate 
R&M  efforts,  tie  R&M  to  operational  goals,  and  ensure 
coordination  across  commands,  systems,  and 
technologies . 

IV.  Establish  a  system  to  ensure  accountability, 
review,  and  feedback  on  the  direction  and  progress  of 
the  R&M  program. 

V.  Establish  a  communication  and  motivation  program 
to  sustain  the  commitment  to  and  organizational 
support  for  the  R&M  improvement  effort. 

VI.  Establish  industry  commitment  to  R&M  to  ensure 
contractors  have  the  motivation  and  capability  to 
support  Air  Force  R&M  requirements.  (16:1) 

The  sixth  is  most  applicable  to  this  thesis.  The  stated 
purpose  of  Objective  VI  is  to  ensure  "internal  planning, 
requirements  documents  and  review  efforts  demand  and  support 
high  priority  for  R&M"  (16:11).  To  achieve  this  objective 
the  Air  Force  must  ensure  "the  specification,  statement  of 
work,  and  proposal  and  evaluation  factors  ace  clear  expres¬ 
sions  of  the  level  of  Air  Force  commitment  and  priority  for 
R&M"  (16:11). 

RADC-TR-79-200 ,  "Reliability  and  Ma intai nabil ity 
Management  Manual "  (2),  is  a  useful  guidebook  which  covers 
reliability  and  maintainability  elements  of  a  complete 


program  from  concept  to  deployment.  It  is  a  general  guide 
but  thorough  in  its  treatment  of  all  the  various  applica¬ 
tions  for  reliability  throughout  the  life  cycle  of  an 
acquisition  program. 

Lessons  Learned  Bulletin  on  Reliability  and 
Maintainability  (9),  published  by  Air  Force  Acquisition 
Logistics  Center  (AFALC) ,  is  a  compilation  of  various 
programs  that  have  been  found  to  be  deficient  in  various 
areas  including  R&M .  Anyone  about  to  embark  on  an  acquisi¬ 
tion  project  should  peruse  this  publication  so  as  not  to 
encounter  similar  mistakes. 

The  Computer  Generated  Acquisition  Documents  System 
(CGAD3)  (20)  was  developed  by  ESD  to  help  simplify  and 
standardize  the  contracting  process  for  ESD  in  addition  to 
implementing  ESDR  800-5.  Tne  developers  of  CGADS  state  that 

levying  a  task  by  citing  a  document  (E.G.,  MIL-SPEC) 
and  paragraph  numbers  is  sufficient.  Citing  a  data 
item  in  parantheses;  E.G.,  (DI-A-1001)  is  all  that  is 

allowed  for  stating  that  a  report  is  required.  Do  not 
add  preparation  and  other  delivery  requirements! 

(20:1) 

The  CGaDS  system  gives  a  good  format/outline  for  a  contract. 
The  system  lists  most  of  the  references  and  DID's  that  may 
oe  cited,  while  not  attempting  to  specify  the  details  for 
each  required  reliability  task.  The  instructions  state  that 
tailoring  ol  the  applicable  task  is  necessary  and  leaves 
that  aspect  up  to  the  or iginator/wr iter  of  the  contract. 

Product  Assurance  and  Test  Engineers  Contracting 
Handbook  (25)  is  a  guide  intended  for  use  by 
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Product  Assurance  and  Test  (PA&T)  Engineers  with 
guidance  and  implementation  materials  for  use  in 
the  preparation  of  tailored  Reliability .. .work 
statements  appropriate  for  incorporating  in  con¬ 
tracts  and  RFP's  (25:ii). 

Overall  this  document  was  found  to  be  useful  and  is 
recommended  for  use  by  anyone  tasked  with  writing 
reliability  requirements  in  a  SOW  (with  slight  grammatical 
rearrangement) .  This  is  the  document  that  prompted  the 
change  of  focus  for  this  thesis. 

MI L-HDBK-338 ,  Electronic  Reliability  Design  Handbook 
is  an  extensive  compilation  of  reliability  oriented 
information. 

This  handbook  describes  a  comprehensive  methodology 
covering  all  aspects  of  electronic  system  reliability 
design  engineering  and  cost  analysis  as  they  relate  to 
the  design  acquisition  and  deployment  of  DOD 
equipment/systems  (12:iii). 

Section  12  of  the  handbook,  the  area  of  most  interest  for 
this  thesis,  discussed  the  specific  program  tasks  recom¬ 
mended  by  MIL-STD-785B.  However  the  discussion  is  limited 
in  that  each  task  is  explained  simply  by  several  exerpts 
directly  from  785B.  But  the  section  titled  "Quantitative 
example  of  the  use  of  weighting  criteria  to  determine  rela¬ 
tive  program  emphasis"  (12:  Sec  12,  39)  was  quite  useful  and 
used  as  a  model  for  the  tables  developed  for  this  thesis. 
This  area  included  Information  not  covered  in  other  publica¬ 
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•  MIL-STD-785B  Reliability  Tasks 

This  chapter  discusses  each  task  of  MIL-STD-735B .  The 
tasks  are  described  in  relation  to  their  purposes. 

Task  101  -  Reliability  Program  Plan 

Task  101  requires  the  contractor  to  provide  a  plan  for 
his  Reliability  Program.  The  plan  shows  that  the  contractor 
understands  the  reliability  requirements  and  describes  how 
he  intends  to  design  and/or  build  a  reliable  item. 

Task  102  -  Monitor/Control  of  Subcontractors  and  Suppliers 

Task  102  requires  the  prime  contractor  to  ensure  that 
his  subcontractors  and  parts  suppliers  will  meet  reliability 
requirements  and  that  provisions  are  made  for  surveillance 
of  their  reliability  activities.  The  prime  contractor  needs 
insight  into  what  the  subcontractors  are  doing  for  the  final 
reliability  is  the  prime's  responsibility. 

Task  103  -  Program  Reviews 

Task  103  concerns  formal  reviews  and  assessments  of 

contract  reliability  requirements.  It  states  specific 

aspects  of  the  program  that  should  be  addressed  at  each  of 

the  live  major  types  of  reviews: 

Preliminary  Design  Review  (PDR) 

Critical  Design  Review  ( CDR) 

Reliaoility  Program  Reviews 
Test  Readiness  Review 
Production  Readiness  Review. 
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Usually  the  reliability  reviews  are  included  as  part  of 
major  reviews  of  the  program  and  is  so  stated  in  the 
Statement  of  Work  (SOW) . 


Reviews  should  reveal  any  problems  that  may  be  present 
in  the  program  and  keep  the  program  manager  updated  so  that 
he  is  assured  all  contractual  reliability  requirements  are 
being  met. 

If  an  acquisition  program  has  reliability  aspects, 
then  some  reliability  reviews  should  be  included.  For 
critical  systems,  reliability  is  an  essential  issue  at  both 
the  Preliminary  Design  Review  (PDR)  and  the  Critical  Design 
Review  (CDR) ,  and  thus  this  task  would  be  a  requirement 
starting  with  FSED  or  earlier. 

Task  104  -  Failure  Reporting ,  Analysis,  and  Corrective 
Action  System  (FRACAS) 

The  FRACAS  requires  a  contractor  to  develop  a  "closed 
loop  system"  (12:  Sec  12,  34;  C4:171,210)  that  collects  data 
on  failed  items,  analyses  the  failure  and  documents  the  cor¬ 
rective  action  taken  to  fix  the  failure. 

The  closed  loop  aspect  refers  to  the  tact  that  the 
analysis  of  the  failure,  and  the  corrective  action  are 
monitored  to  ensure  a  timely  and  complete  fix.  A  well 
designed  and  utilized  FRACAS  will  assist  in  reliability 
growth  and  identify  problems  during  Task  302,  Reliability 
Development/Growth  Testing. 


."J 


'V.  . 1  1 . 


Task  105  -  Failure  Review  Board  (FRB) 

The  FRB  is  responsible  for  reviewing  failures, 
assigning  failure  causes,  and  monitoring  the  FRACAS  to 
ensure  failures  are  analyzed  and  the  corrective  action  is 
adequate.  The  contractor  must  appoint  members  to  the  board 
from  design,  reliability,  safety,  manufacturing  and  quality 
assurance  to  investigate  failures.  The  FRB  consists  of 
contractor  personnel  with  an  AF  representative  as  an 
observer . 

To  avoid  redundancy  with  other  quality  functions, 
primarily  the  Material  Review  Board,  "this  task  shall  be 
coordinated  with  Quality  Assurance  organizations  to  insure 
there  is  no  duplication  of  effort"  (18:  Sec  105,  1). 
Normally  all  programs  with  a  FRACAS  also  have  this  task. 

Task  201  -  Reliability  Modeling 

A  reliability  model  is  a  mathematical  equation  which 
defines  the  relationship  between  the  failure  rate  of 
an  assembly  (equipment,  or  system)  and  the  failure 
rate  of  all  the  parts  which  make  up  the  assembly 
(1:46) . 

The  task  requires  the  contractor  to  develop  and  update 
reliability  models  using  techniques  defined  in  MIL-HDBK- 
217.  Models  provide  the  framework  for  the  allocations  and 
predictions  (Tasks  202  and  203)  of  reliability. 

Task  202  -  Rel iaoility  Allocations 

Reliability  allocation  apportions,  or  gives,  to  each 
subsystem  a  reliability  goal  based  on  the  total  system 
reliability  requirement. 
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Requiring  this  task  can  be  helpful  for  establishing 
baseline  requirements  for  subsystem  designs.  Since  the  prime 
contractor  does  not  usually  supply  every  subsystem,  this 
task  also  establishes  the  reliability  requirement  for 
subcontracted  material.  Any  parts  supplied  by  government 
furnished  equipment  (GFE) ,  vendors  or  subcontractors  need  to 
conform  to  the  total  system  reliability. 

Task  203  -  Reliability  Predictions 

A  reliability  prediction  is  an  estimate  of  the 
reliability  of  your  system,  based  on  historical  data  from 
similar  parts,  assemblies  or  systems.  The  longer  the 
component  has  been  in  operation  and  failure  rate  data  has 
been  collected,  the  better  future  predictions  will  be.  The 
reliability  prediction  of  Task  203  requires  the  contractor 
to  develop  predictions  of  whole  assembly  reliability  based 
on  the  failure  rates  of  component  parts.  However  if  failure 
rates  have  not  yet  been  determined,  as  in  the  conceptual 
phase,  and  many  unique  parts  are  to  be  used,  (not  a  good 
idea  -  as  many  off  the  shelf/GFE  parts  as  possible  should  be 
used)  other  predictive  techniques  need  to  be  developed. 

Predictions  and  allocations  are  used  by  Logistics 
Support  Analysis  (LSA)  in  order  to  determine  the  required 
number  of  spares,  maintenance  people,  facilities,  and  life 
cycle  cost. 


(FMECA) 

As  a  result  of  Task  204  the  designer  determines  pos¬ 
sible  failures  of  a  design  and  the  effects  of  those  failures 
on  the  total  system  performance.  FMECA  is  "a  companion 
effort  along  with  reliability  modeling,  prediction,  allo¬ 
cation  and  design"  (2:48).  If  possible  problems  are  identi¬ 
fied  early  by  a  FMECA,  the  corrective  design  changes  can  be 
easily  implemented  with  least  cost. 

Task  205  -  Sneak  Circuit  Analysis 

A  Sneak  Circuit  Analysis  (SCA)  determines  the  various 
paths  a  circuit  might  take  and  what  effects  on  the  system 
unwanted  signals  might  make.  An  example  of  a  sneak  circuit 
might  be:  a  circuit  designed  to  raise  the  landing  gear  will 
inadvertently  be  triggered  if  the  pilot  were  to  turn  on  all 
the  lights  in  a  certain  order.  This  is  a  possibly  disas¬ 
trous  occurrence  if  the  aircraft  were  traveling  down  the 
runway . 

SCA  is  expensive  to  accomplish,  therefore,  should  be 
reserved  for  critical  systems:  flight  and  weapon  controls, 
space  application,  etc.  in  order  to  identify 

designeo-in  conditions  that  could  inhibit  desired 
system  functions  or  produce  undesired  system  functions 
wnicn  could  adversely  affect  crew  safety  or  mission 
equipment  (1 :  iii) . 


RADC-TR-82-179 ,  "Sneak  Analysis  Application  Guide¬ 


lines"  (1)  is  a  comprehensive  report  derived  from  the 
investigation  of  numerous  projects  where  SCA  haa  been 
applied.  This  report  gives  guidelines  about  when  and  on 
what  types  of  systems  SCA  should  be  applied. 

A  SCA  should  be  done  as  early  as  possible  in  the 
acquisition  cycle,  but 

Since  Sneak  Analysis  is  based  on  detailed  system 
drawings  and  computer  program  instructions,  the  most 
likely  early  phases  for  implementation  would  be  the 
Full-Scale  Engineering  Development  and  Full-Scale 
Prototype  Development  phases  (1:6). 

If  sneak  circuit  problems  can  be  identified  early,  design 

changes  are  easier  and  less  costly  to  implement. 

Task  206  -  Electronic  Parts/Circuits  Tolerance  Analysis 

An  electronic  tolerance  analysis  analyzes  the  effects 
that  design  parameters  of  a  system  will  have  on  electronic 
parts  and  circuits.  A  good  design  will  be  tolerant  of  a 
range  of  parameters.  Since  a  great  many  factors  must  be 
taken  into  consideration,  Task  206  is  costly  and  recommended 
only  tor  critical  systems. 

Task  207  -  Parts  Program 

Task  207  requires  the  contractor  to  develop  a  compre¬ 
hensive  program  that  identifies  all  parts  and  controls  the 
acquisition  of  parts  so  that  reliability  is  assured.  The 
stated  objective  of  this  task  is  "to  control  the  selection 
and  use  of  standard  and  nonstandard  parts"  (18:A12)  with  the 


underlying  objective  being  "to  ensure  that  the  contractor 
will  select,  whenever  possible,  standard  military  parts  of 
the  preferred  type  and  quality"  (19:67).  Most  military 
parts  are  already  tested,  and  approved.  Using  nonstandard 
parts  can  increase  the  acquisition  cost  of  a  program  because 
of  the  need  for  their  design,  test  and  approval.  In  addi¬ 
tion,  nonstandard  parts  increase  the  number  of  spare  parts 
maintained  in  the  Air  Force  inventory.  A  parts  program  is 
expensive,  however  if  the  program  is  effective,  the  reduced 
LCC  will  compensate  for  the  original  investment. 

Task  208  -  Reliability  Critical  I  terns 

Task  208  requires  the  contractor  to  identify  the 
subsystems/components  that  are  so  critical  that  their 
failure  would  cause  the  whole  system  to  fail.  The  Task  also 
specifies  the  necessity  for  the  identification  of  "methods 
and  procedures  for  control  and  testing  of  (those)  relia¬ 
bility  critical  items"  (18:  Sec  208,  1).  The  inputs  to  this 
task  come  from  the  FMECA  or  other  analysis. 

Task  209  -  Effects  of  Functional  Testing ,  Storage ,  Handling, 
Packaging ,  Transportation,  and  Maintenance  (PHT) 

Task  209  requires  the  contractor  to  develop  procedures 
to  identify  what  effects  PHT  will  have  on  the  system, 
formally  this  item  should  be  handled  by  PHT  engineers  and 
care  needs  to  be  taken  that  the  contractor  does  not  charge 
twice  for  the  same  work  if  this  task  is  required. 
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Task  301  -  Environmental  Stress  Screening  (ESS) 

ESS  requires  the  contractor  to  "establish  and  imple¬ 
ment.  .  .procedures"  (18:  Sec  301,  1)  to  identify  and  remove 
bad  parts  and  workmanship.  This  'burn-in'  usually  consists 
of  short  duration  tests  that  severely  stress  a  component 
beyond  its  expected  normal  operating  environment  with  the 
intent  to  stimulate  failures.  Parts,  subassemblies  and 
complete  units  are  subjected  to  shock,  vibrations,  extreme 
heat,  cold,  dust,  sand,  and  nuclear  radiation  under  the  most 
severe  situations  that  components  would  ever  be  expected  to 
encounter . 

Regardless  of  time  or  cost,  environmental  testing  is 
an  essential  part  of  any  reliability  program  and  not  a 
safe  place  to  look  for  'money-saving'  shortcuts.  The 
life  cycle  cost  penalty  can  dwarf  any  development  cost 
saving.  (2:62) 

Task  302  -  Reliability  Development/Growth  Test  (RDGT) 

Program 

Usually  the  initial  reliability  predictions  are  not 
achieved  in  the  prototype  or  even  during  FSED.  By  using  the 
RDGT  concept,  failures  are  expected.  When  they  occur,  the 
failures  are  analyzed  and  corrective  action  is  taken  as 
applicable.  This  testing,  finding  and  correcting  of  fail¬ 
ures  will  improve  reliability  over  time  (this  is  reliability 
growtn) . 
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Task  303  -  Reliability  Qualif ication  Test  (RQT)  Program 

Task  303  requires  the  contractor  prove  the  system  is 
ready  for  production,  in  that  it  meets  the  'minimum' 
reliability  requirement.  The  RQT  is  a  one  time  test  to 
demonstrate  the  reliability  required  in  the  specification. 

Task  304  -  Production  Reliability  Acceptance  Test  (PRAT) 
Program 

The  PRAT  program  is  an  ongoing  effort  throughout 
production.  The  program  consists  of  periodically  testing, 
in  a  simulated  operational  environment,  components  from 
production.  The  test  is  to  verify  that  the  production  units 
continue  to  meet  reliability  requirements. 
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.  Cost  Effectiveness  and  Appl icat ion  Analysis 

This  chapter  discusses  the  cost  effectiveness  (C/E)  of 
the  reliability  tasks  in  MIL-STD-785  and  discusses  the 
application  of  the  tasks  according  to  program  phase.  Two 
examples  of  how  to  determine  the  relative  C/E  of  the  tasks 
for  a  particular  acquisition  program  are  presented.  The 
question  of  whether  to  require  a  contractor  to  accomplish 
the  tasks  is  subjectively  answered. 

Background 

The  primary  reason  any  reliability  task  should  be 
required  is  to  acquire  a  reliable  system.  However  every 
acquisition  program  has  cost  constraints,  and  each  extra 
requirement  on  the  contractor  exacts  a  cost.  Therefore  some 
sort  of  cost  analysis  should  be  performed. 

A  C/E  influences  table  from  MIL-HDBK-338  (12:  Sec  12, 
4d)  was  used  as  a  model  for  analysis  of  the  cost  effective¬ 
ness  of  MIL-STO-735B  tasks.  The  table  is  used  to  rate 
relative  cost  effectiveness  of  a  task  (on  a  scale  from  1  to 
5)  based  on  the  following  major  criteria;  complexity  of 
design,  equipment  criticality,  quantity  purchased,  equipment 
operating  environment,  level  of  technology  and  storage 
requirements.  Each  major  criterion  is  scaled  according  to 
two  factors.  For  example,  complexity  is  either  high  or  low, 
technology  is  standard  or  state  of  the  art. 
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Empty  tables  were  given  to  AF  Institute  of  Technology 
(AFIT)  reliability  instructors  and  to  reliability  engineers 
in  Aeronautical  Systems  Division  (ASD)  and  AF  Acquisition 
Logistic  Center  (AFALC)  with  instructions  that  they  were  to 
subjectively  fill  in  the  numbers,  ie.  how  they  'felt'  the 
cost  effectiveness  was  for  each  task.  Tables  I  through  III 
are  the  averaged  results  of  the  responses.  The  tables  show 
a  wide  diversity  of  opinion  as  to  the  cost  effectiveness  of 
many  of  the  individual  tasks.  Some  respondents  had  mostly 
very  low  numbers,  while  one  respondent  hardly  discriminated 
at  all,  giving  a  preponderance  of  fives  to  most  of  the 
tasks.  Evidently  he  was  of  the  opinion  all  the  tasks  under 
every  criteria  are  very  cost  effective. 

Cost  E£  Eectiveness  Analysis 

In  this  section  each  of  the  three  tables  are  dis¬ 
cussed.  The  discussion  focuses  on  the  average  numbers  as 
shown  by  the  tables.  The  method  for  general  evaluation  of 
the  tables  was  to  sum  all  the  numbers  for  each  task  (across 
the  tables) .  The  highest  sum  thus  indicated  that  task  to  be 
most  cost  effective  (C/E) .  The  lowest  sums  were  indicative 
of  those  tasks  least  C/E.  Certain  criteria  (down  the 
columns)  which  showed  a  significant  consensus  were  also 
discussed  (ie.  Hi  Complexity  for  space  and  missile  systems. 
Table  I,  has  almost  all  4's  and  5's). 

Table  I:  Cost  Effectiveness  Influences  For  Space  and 


space  and  missile  systems  are  Tasks  104  and  105,  Failure 
Reporting,  Analysis  and  Corrective  Action  System  (FRACAS) 
and  Failure  Review  Board  ( FRB) .  Tasks  207  and  301,  Parts 
Program  and  ESS,  also  have  relatively  high  scores.  The 
emphasis  seemed  to  be  placed  on  controlling  part  reliabil¬ 
ity,  through  the  parts  program,  screening  the  parts  and  then 
using  failure  data  to  highlight  the  remaining  problems. 

The  Reliability  Program  Plan,  Task  101  has  the  lowest 
total  average  score,  with  Tasks  205,  303  and  304  (Sneak 
Circuit  Analysis  (SCA) ,  Reliability  Qualification  Test 
(RQT) ,  and  Production  Reliability  Acceptance  Test  PRAT)) 
also  having  low  numbers. 

It  is  also  apparent  that  it  would  be  considered  cost 
effective  to  require  almost  all  of  the  tasks  for  items  of 
high  complexity  and  criticality  using  technology  that  is 
pushing  the  state  of  the  art  (SOA) . 

Requiring  a  program  plan  is  primarily  a  paperwork 
exercise  and  if  cost  is  indeed  a  constraint  (every  piece  of 
paper  required  costs  upward  of  $50)  then  this  task  may  be 
the  first  to  be  eliminated.  The  RQT  may  not  be  cost 
effective  but  is  usually  required  (called  a  Qual  test)  for 
space  systems  and  there  is  at  least  one  occasion  where  the 
Qual  test  satellite  was  actually  sent  into  orbit.  The  PRAT 
is  another  test  that  is  nice  to  have  but  may  not  be  cost 
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COST  EFFECTIVENESS  INFLUENCES 
FOR  SPACE  AND  MISSILE  SYSTEMS 
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effective.  However  when  it  is  not  required,  the  production 
items  will  be  operationally  tested  in  the  field  with 
possibly  disastrous  results. 

The  cost  effectiveness  of  SCA  does  not  rate  very  high 
in  all  three  tables.  If  a  SCA  is  required,  critical  systems 
need  a  more  sophisticated  and  complete  analysis  then  less 
critical  systems;  however,  the  more  parts  and  paths  involved 
the  higher  the  cost.  Figure  1  gives  a  rough  estimate  in 
graphical  form  of  the  escalating  cost  of  a  SCA  according  to 
the  number  of  parts. 


Figure  1.  Relative  Cost  of  SCA 
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Table  II :  Cost  Ef f ect iveness  Influences  For  Avionics 
Electrical  Equipment .  Environmental  Stress  Screening  (ESS) 
and  a  Parts  Program  (Tasks  301  and  207)  seem  to  be  thought 
most  cost  effective  across  the  board  while  Tasks  209  and  103 
(Effects  of  Functional  Testing,  Storage,  Handling,  Packag¬ 
ing,  Transportation,  and  Maintenance  (PHT)  and  Program 
Reviews)  least  C/E.  Again  complex  and  critical  systems  have 
high  scores  for  almost  all  tasks.  Pushing  the  SOA  does  not 
seem  to  have  as  much  weight  as  it  does  for  space  and  missile 
sys  terns . 

Usually  program  reviews  take  up  valuable  time  both 
from  the  contractor  and  AF  program  management  staffs.  The 
reliaoility  reviews  should  be  incorporated  with  total  system 
reviews.  If  the  reviews  are  planned  for  in  advance,  with 
data  being  accumulated  and  peridically  analyzed  with 
reviews  in  mind,  they  snould  not  require  an  extraordinary 
effort. 

Some  engineers  believe  Task  209  (PHT)  should  not 
really  fall  under  the  responsibility  of  reliability  but 
belongs  in  another  area. 
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Table  III ;  Cost  Effectiveness  Influences  For 
Airframe/  Mechanical  Equipment .  Program  Reviews,  FRACAS  and 
a  Parts  Program  (TasKs  103,  104  and  207)  were  determined  to 
be  most  cost  effective  for  airframe/mechanical  equipment. 
Tasks  205  and  206  (SCA  and  Electronic  Parts/Circuits  Toler¬ 
ance  Analysis)  were  found  to  be  least  C/E.  Since  Tasks  205 
and  206  deal  with  electronic  systems,  it  seems  logical  that 
these  two  tasks  would  not  be  C/E  for  mechanical  or  airframe 
equipment. 

The  three  criteria:  high  complexity,  high  equipment 
criticality  and  SOA  Technology  continue  to  rate  higher  for 
most  of  the  Tasks  then  do  the  other  criteria,  but  in 
relation  to  the  two  previously  discussed  charts  there  is  an 
overall  lower  ranking. 


COST  EFFECTIVENESS  INFLUENCES 
FOK  AIRFRAME/MECHANICAL  EQUIPMENT 
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Detecminat ion  of  Task  Cost  Effectiveness 

The  following  few  pages  will  show  two  examples  of  how 
the  C/E  tables  can  be  used.  The  first  step  in  using  the 
tables,  is  to  categorize  a  piece  of  equipment  according  to 
the  criteria.  Figure  2  is  an  example  of  categorizing  a 
particular  piece  of  equipment.  In  this  case  a  cockpit 
lighting  system. 

Cockpit  Lighting  System 


Complexity 

— 

Low 

Equip  Criticality 

- 

Low 

Quantity 

- 

Hi 

Equip  Oper.  Environment 

- 

Benign 

Technology 

- 

Standard 

Storage  Time 

- 

Short 

Figure  2.  Categorization  of  Equipment 

A  matrix  is  then  developed  (Table  IV)  by  taking  the 
appropriate  numbers  from  the  applicable  cost  effectiveness 
table  for  each  criteria  and  each  task.  Because  this  is 
electrical  equipment,  Table  II  is  used.  Each  of  the  numbers 
for  a  task  is  multiplied  to  determine  a  total  weighting 
factor  (ie.  for  Task  101  multiply  2X2X3X2X2X2  = 

96).  Using  a  scale  such  as  20  (or  10,  30  etc.),  the  highest 
number,  972,  is  given  a  weighting  factor  of  20.  The  next 
subsequent  lower  number  is  divided  by  the  highest  (486/972  = 
.5)  and  multiplied  by  20  (20  X  .5  =  10)  to  get  a  weighting 
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Table  IV 


Cockpit  Lighting  C/E  Matrix 


Task 

Comp 

Lo 

Cr  it 

Lo 

Quan 

Hi 

Envir 

Benign 

Tech 

Std 

Store 

Shor  t 

Total 

Facto 

101 

2 

2 

3 

2 

2 

2 

96 

2 

102 

2 

2 

4 

2 

3 

2 

192 

4 

103 

2 

2 

3 

2 

2 

2 

96 

2 

104 

3 

3 

4 

3 

3 

3 

972 

20 

105 

2 

2 

4 

3 

3 

3 

432 

9 

201 

2 

2 

3 

2 

2 

2 

96 

2 

202 

2 

2 

3 

3 

3 

3 

324 

6.7 

203 

3 

3 

3 

3 

3 

2 

486 

10 

204 

2 

2 

3 

2 

3 

3 

216 

4.4 

205 

1 

1 

3 

2 

2 

2 

24 

.5 

206 

2 

2 

3 

1 

2 

2 

48 

1 

207 

3 

3 

4 

3 

3 

3 

972 

20 

208 

2 

2 

3 

2 

2 

2 

96 

2 

209 

2 

2 

3 

2 

2 

2 

96 

2 

301 

3 

3 

4 

3 

3 

3 

432 

9 

302 

2 

2 

4 

2 

3 

2 

192 

4 

3  0  3 

2 

2 

3 

2 

3 

2 

144 

3 

i  304 

2 

2 

4 

2 

3 

2 

192 

4 
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factor  of  10.  The  rating  for  each  task  is  achieved  the  same 
way. 

The  major  emphasis  for  the  lighting  system  should  be 
on  Tasks  104  and  207  (FRACAS  and  Parts  Program)  as  they  have 
the  highest  weighting  factor  of  20.  Tasks  203,  105  and  301 
(Reliability  Predictions,  FRB  and  ESS)  might  also  be 
considered  with  their  factors  of  10,  9  and  9. 

The  least  C/E  tasks  would  be  205  and  206  (SCA  and 
Electronic  Parts/Circuits  Tolerance  Analysis) .  For  a  simple 
system  those  two  tasks  would  most  likely  cost  much  more  than 
they  are  worth.  Tasks  101,  103,  201,  208  and  209  (Program 
Plan,  Program  Reviews,  Reliability  Modeling,  Reliability 
Critical  Items,  and  PHT)  also  rate  low  with  factors  of  2. 
These  last  five  tasks  are  paperwork  exercises  that  can  be 
useful  for  critical  systems  but  may  not  be  C/E  for  simple 
systems . 

Another  example  of  using  the  C/E  tables  might  be  for  a 
complex  airborne  missile  system.  Figure  3  shows  the  cate¬ 
gorization  by  criteria  and  Table  V  gives  a  C/E  matrix  for 
this  missile  system.  For  this  example  Table  I,  Cost  Effec¬ 
tiveness  Influences  for  Space  and  Missile  Systems,  is  used. 
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Airborne  Missile  System 


Complexity  -  Hi 

Equip  Criticality  -  Hi 

Quantity  -  Low 

Equip  Oper.  Environment  -  Hostile 

Technology  -  SOA 

Storage  Time  -  Long 


Figure  3.  Categorization  of  Equipment 

As  in  the  previous  example  each  of  the  numbers  for  a 
task  are  multiplied  to  determine  a  total  weighting  factor 
(ie.  for  Task  101  multiply  3X4X2X3X3X3=  643) .  A 
rating  of  20,  is  given  to  the  highest  number,  4800.  The 
next  subsequent  lower  number  is  divided  by  the  highest 
(3600/4800  =  .75)  and  multiplied  by  20  (20  X  .75  =  15)  to 
get  a  rating  of  15.  The  process  is  continued  for  each  task 
to  get  the  lower  factors. 

The  major  emphasis  for  the  Airborne  Missile  System 
should  be  on  Tasks  104,  105  and  301  (FRACAS,  FRB  and  ESS)  a 
they  have  the  highest  weighting  factors  of  20,  20,  and  15. 
Tasks  204  and  207  (FMECA  and  Parts  Program)  also  score 
fairly  high  with  factors  of  12  and  10  respectively. 
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It  should  be  remembered  that  Tables  I  through  III  are 
composites  of  subjective  numbers  applied  according  to  the 
various  criteria  for  the  different  reliability  tasks.  These 
numbers  were  gathered  from  experts  in  reliability  working  at 
Wright  Patterson  AFB,  each  of  whom  has  his  own  biases. 

These  numbers  and  criteria  are  subjective  and,  as  MIL-HDBK- 
333  states,  "each  manager  may  want  to  develop  his  own 
weignting  factors  criteria  and  the  specific  numbers  for  his 
given  application"  (12:Sec  12,  41). 

Appl ication  Analysis 

In  this  section  the  four  major  acquisition  phases  and 
the  applicability  of  the  reliability  tasks  within  each 
phase  are  discussed. 

The  tools  used  for  the  task  applicability  analysis 
are  the  Application  Matrix  (Figure  4)  taken  from  MIL-STD- 
785B  and  the  Reliability  Program  Elements  chart  (Figure  5) 
(available  in  several  publications)  which  shows  criticality 
of  reliability  tasks  in  relation  to  the  acquisition  life 
cycle  phases. 
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Figure  4.  Application  Matrix 
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Reliability  Acceptance  Tests 


xx .  .1 . AAAAAobooooooo 


First  contract  _ 

.  Desirable  activity  (for  highest  success 

probabil  i  ty) 

000000  Necessary  activity  (errors  seldom  disastrous) 

xxxxxx  Very  important  activity  (errors  usually 
disastrous) 

.  tow  key  activity  {to  update  previous  results) 

AAAAAA  Critical  Activity 


FigurG  5.  Reliability  Program  Elements 
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Concept  Phase .  The  concept  phase  is  the  time  during 
which  alternatives  are  proposed  to  meet  a  particular  threat 
and  satisfy  a  particular  need.  Often  more  than  one  contrac¬ 
tor  will  be  tasked  to  work  on  the  concept,  in  order  to  have 
at  least  two  original  designs  {"competitive  exploration  of 
alternatives"  (23:  Sec  IV,  1). 

Very  few  of  the  reliability  tasks  are  actually  applic¬ 
able  during  this  phase  since  the  design  is  still  being 
formulated.  The  program  manager  must  be  considering  relia¬ 
bility  and  the  contractor  must  also  be  designing  for  relia¬ 
bility,  but  usually  only  tasks  201  and  203,  Reliability 
Modeling,  and  Reliability  Predictions  are  applicable.  These 
models  and  predictions  will  undoubtedly  "be  rather  crude 
during  the  conceptual  phase,  it  (they)  will  be  expanded  and 
refined  as  more  system  details  are  evolved"  (2:46) . 

Reliability  Predictions  were  not  rated  as  very  cost 
effective  on  any  of  the  three  C/E  tables;  however, 
predictions  can  prevent  disastrous  results  as  depicted  by 
the  Reliability  Program  Elements  chart,  Figure  5.  This 
chart  shows  prediction  to  be  a  very  important  activity 
during  the  later  half  of  conceptual  stage,  throughout  valid¬ 
ation  and  into  FSED.  Errors  here  are  usually  disastrous. 

If  predictions  of  reliability  for  component  parts  do  not 
come  true,  then  the  reliability  of  the  whole  system  will  be 


ef  f ected . 


Demon s t ra tion/Validation  Phase . 

In  the  Demonstration  and  Validation  phase,  definiti- 
zation  of  the  selected  alternative (s)  is  expanded,  and 
the  value  and  practicality  of  the  increasingly  speci¬ 
fic  design  approach  continues  to  be  checked  (23:  Sec 
VI,  1)  . 

In  this  phase  a  Program  Plan,  Task  101  could  be 
required  of  the  competitors  so  that  they  could  show  what 
they  plan  to  do  if  awarded  the  contract;  however,  since 
program  plans  are  not  deemed  C/E,  this  task  may  not  be 
required  at  least  on  the  more  cost  constrained  programs. 
Modeling  and  Predictions  would  continue  from  the  Concept 
Phase  and  be  revised  and  updated  as  appropriate.  Relia¬ 
bility  Allocation  (Task  202)  could  begin,  for  the  major 
subsystems  would  now  be  identified.  Also  Reliability 
Critical  Items  (Task  208)  and  FMECA  (Task  204)  could  be 
required,  realizing  further  refinement  would  be  necessary 
during  FSED . 

Full  Scale  Engineering  Development  (FSED) .  The  FSED 
is  the  phase  where  the  'final'  design  is  approved  and  a 
preproduction  prototype  is  built,  tested  and  approved.  It 
is  during  this  phase  that  almost  all  tasks  are  applicable 
(if  they  are  applicable/cost  effective  for  a  particular 
program).  PRAT  (Task  304)  is  generally  not  applicable  since 
this  task  is  specifically  used  to  test  production  items. 

Production  Phase .  The  final  acquisition  phase  is  the 
production  phase  (deployment  and  phase-out  are  not  part  of 
acquisition)  where  the  object  of  the  first  three  phases  is 
produced . 
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The  primary  task  requirement  during  this  phase  would 
be  the  PRAT  program  task  304.  A  limited  program  plan  might 
also  be  required  from  the  contractor  to  update  his  continu¬ 
ing  efforts  for  reliability.  Tasks  102  and  207  (monitoring 
of  subcontractors  and  the  parts  program)  are  also  important 
in  this  phase  to  assure  there  is  a  continuous  supply  of 
reliable  parts  and  subcomponents. 

If  other  200  level  tasks  had  been  required  during 
development,  the  data  and  results  of  tests  required  by  those 
tasks  should  be  reviewed  to  verify  if  the  predictions  cor¬ 
relate  with  actual  operational  reliability  (after  the  first 
lot  from  production  is  operational) .  Similarly  the  results 
from  the  ESS  and  the  RQT  need  to  be  reviewed. 


V. 


Summary ,  Conclusions,  Recommendations 


Summary 

Simply  stating  the  desired/required  reliability  for  a 
system  in  a  contract  does  not  guarantee  reliability.  The 
majority  of  individuals  questioned  by  the  R&M  2000  working 
group  were  of  the  opinion  that  management  attention  was  of 
prime  importance.  Evidently,  management,  both  Air  Force  and 
contractor,  must  wholeheartedly  support  and  follow  through 
on  a  reliability  program  for  it  to  be  successful. 

While  researching  various  contracts  for  this  paper,  a 
contract  was  found  consisting  of  at  least  75  pages  of  speci¬ 
fications,  SOW  and  CDRL  pertaining  to  the  purchase  of  a 
shipping  container  for  some  part.  In  other  words,  they 
basically  wanted  to  bay  a  cardboard  box.  This  author 
believes  the  extraordinary  paperwork  and  micro-management 
can  oe  reduced/eliminated.  Indeed  it  is  necessary  for  the 
Air  Force  to  specify  what  is  needed,  and  toe  conditions  and 
time  under  which  the  systems  must  operate;  but  increasing 
the  lengths  of  contracts  lengthens  the  procurement  period 
and  increases  the  acquisition  cost. 

This  research  effort  focused  on  the  cost  effectiveness 
of  the  tasks  of  MIL-STD-785B  and  discovered  several  tasks 
thought  to  be  not  cost  effective  (relative  to  other  tasks). 
The  elimination  of  some  of  those  tasks  (if  in  fact  the 
elimination  of  the  tasks  would  indeed  not  adversely  effect 
the  reliability  of  the  final  product)  from  contracts  may 
help  to  speed  up  the  acquisition  process  and  reduce  costs. 


Conclusions 


The  relatively  wide  range  of  numbers  applied  to  the 
cost  effectiveness  tables  by  the  different  engineers 
reflects  a  diversity  of  opinion  that  could  significantly 
effect  acquisition  programs.  The  number  of  tasks  included 
in  a  contract  depends  on  the  opinions  of  the  writer 
(reliability  engineer)  of  that  contract.  One  author  may 
wholeheartedly  believe  every  task  is  important  (and 
therefore  cost  effective)  and  include  more  tasks  than 
absolutely  necessary  to  get  the  desired  reliability. 

Another  writer  may  not  think  many  of  the  tasks  are  of  value 
an’  he  may  possibly  leave  out  a  necessary  task(s). 

The  cost  effectiveness  method  of  rank  ordering 
reliability  tasks  is  only  one  method  of  determining  what 
should  be  included  in  a  contract.  This  method  can  help 
managers  prioritize  areas  for  the  managing  of  time  and 
money.  It  is  only  one  method,  and  if  used,  it  needs  to  be 
used  in  conjunction  with  common  sense  and  other  methods  that 
may  be  available. 

Having  found  Columbia  Research  Corporation's  hand¬ 
book,  which  is  a  worthwhile  guide  to  contracting  for 
reliability,  a  question  arises:  why  isn't  it  being  used? 
When  this  project  began,  it  was  thought  by  several 
experienced  persons  that  there  was  not  available  a  truly 
useful  guide  on  how  to  contract  for  reliability.  Since  1982 
Columbia's  handbook  has  been  available  with  little  notice. 
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Recommendations 


A  possible  follow-on  topic  to  this  thesis  is  the 
investigation  of  procurements,  to  discover  if  there  are  any 
contracts  with  a  minimal  amount  of  specifications  and 
requirements  and  what  (if  any)  deleterious  effects  there  are 
on  reliability  and  effectiveness  of  the  system.  Or  the 
reverse  case  could  be  investigated.  A  contract  (or  con¬ 
tracts)  with  excessive  specifications  and  requirements  could 
be  the  object  of  research  to  determine  if  those  excesses 
helped  get  a  better,  more  reliable  product  or  in  fact  were 
only  a  hindrance.  The  actual  cost  of  each  specification  and 
requirement  could  be  determined  and  some  correlation,  of 
actual  cost  vs.  reliability  acquired,  might  be  obtained 
In  line  with  the  underutilization  of  publications 
paid  for  by  DOD  and  dealing  with  the  government,  a  topic 
might  be  to  investigate  such  publications  and  determine  the 
actual  usage  (versus  cost)  of  those  publ icat ions . 


Appendix:  Acronyms 


AFALC  -  Air  Force  Acquisition  logistics  Center 
AFSC  -  Air  Force  Systems  Command 

AFOTEC  -  Air  Force  Operational  Test  &  Evaluation  Center 

ASD  -  Aeronautical  Systems  Division 

AFTEC  -  Air  Force  Test  &  Evaluation  Center 

C/E  -  Cost  Effectiveness 

CDR  -  Critical  Design  Review 

CDRL  -  Contract  Data  Requirements  List 

CERT  -  Combined  Environment  Reliability  Test 

DOD  -  Department  of  Defense 

DODD  -  Department  of  Defense  Directive 

ESD  -  Electronic  Systems  Division  (AFSC) 

ESS  -  Environmental  Stress  Screening 

FMECA  -  Failure  Modes  and  Criticality  Analysis 

FRACAS  -  Failure  Reporting,  Analysis  and  Corrective 

Action  System 

FRB  -  Failure  Review  Board 

ESED  -  Full  Scale  Engineering  Development 

GFE  -  Government  Furnished  Equipment 

LSA  -  Logistics  Support  Analysis 

LCC  -  Life  Cycle  Costs 

MTBf  -  Mean  Time  Between  Failure 

MTBM  -  Mean  Time  Between  Maintenance 

MIL-HDBK  -  Military  Handbook 

MIL-STD  -  Mii..tary  Stanuard 
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PA&T 


Product  Assurance  and  Test 


PHT 

Effects  of  Functional  Testing,  Storage,  Hand¬ 
ling,  Packaging,  Transportation  and  Maintenance 

PDR 

- 

Preliminary 

Design  Review 

PRAT 

- 

Production  Reliability  Acceptance  Test  Program 

RADC 

- 

Rome  Air  Development  Center 

RFP 

- 

Request  for 

Proposal 

RDGT 

- 

Reliability 

Development/Growth  Test 

RIW 

- 

Rel iabi 1 i ty 

Improvement  Warranty 

R&M 

- 

Reliability 

and  Maintainability 

RQT 

- 

Reliability 

Qualification  Test 

SCA 

- 

Sneak  Circuit  Analysis 

SON 

- 

Statement  of 

Need 

SOW 

— 

Statement  of 

Work 
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